Papers with architectural modifications
WISCA: A Lightweight Model Transition Method to Improve LLM Training via Weight Scaling (2026.findings-acl)
Copied to clipboard
Jiacheng Li, Jianchao Tan, Zhidong Yang, Pingwei Sun, Feiye Huo, Jiayu Qin, Xiangyu Zhang, Maoxin He, Guangming Tan, Weile Jia, Xunliang Cai, Tong Zhao
| Challenge: | Recent advances in training optimization for Transformer-based large language models lack systematic optimization of weight patterns during training. |
| Approach: | They propose a Weight Scaling method that rescales weights while preserving model outputs to improve model training efficiency and model quality. |
| Outcome: | The proposed method significantly improves convergence quality and loss reduction in LLMs with Grouped Query Attention architectures and LoRA fine-tuning tasks. |
Sparse Latents Steer Retrieval-Augmented Generation (2025.acl-long)
Copied to clipboard
Chunlei Xin, Shuheng Zhou, Huijia Zhu, Weiqiang Wang, Xuanang Chen, Xinyan Guan, Yaojie Lu, Hongyu Lin, Xianpei Han, Le Sun
| Challenge: | In this study, we uncover interpretable latents that govern RAG behavior in large language models . Sparse Autoencoders are used to control large language model (LLM) behavior . |
| Approach: | They leverage Sparse Autoencoders within the LLaMA Scope to uncover latents that govern RAG behaviors. |
| Outcome: | The proposed model can be used to control large language models without architectural modifications. |
KILM: Knowledge Injection into Encoder-Decoder Language Models (2023.acl-long)
Copied to clipboard
| Challenge: | Large pre-trained language models retain implicit knowledge within their parameters, but are susceptible to memorizing the pretraining corpora rather than capturing the knowledge within them. |
| Approach: | They propose to inject entity-related knowledge into encoder-decoder PLMs via a generative knowledge infilling objective through continued pre-training. |
| Outcome: | The proposed approach outperforms state-of-the-art models on general NLU and NLG tasks while maintaining their original performance. |
Segment Any Text: A Universal Approach for Robust, Efficient and Adaptable Sentence Segmentation (2024.emnlp-main)
Copied to clipboard
| Challenge: | Sentence segmentation is a key task in many NLP systems, but no prior method has achieved all of the features needed to segment a text into sentences. |
| Approach: | They propose a new model that uses punctuation to enhance robustness and adaptability. |
| Outcome: | The proposed model outperforms baselines across 8 corpora across diverse domains and languages and is available under the MIT license. |
CARMA: Enhanced Compositionality in LLMs via Advanced Regularisation and Mutual Information Alignment (2025.emnlp-main)
Copied to clipboard
| Challenge: | Large language models (LLMs) struggle with compositional generalisation, limiting their ability to systematically combine learned components to interpret novel inputs. |
| Approach: | They propose a method that enhances stability and robustness of compositional reasoning in large language models (LLMs) by regularising and preserving layer-wise stability constraints. |
| Outcome: | The proposed method reduces variability, stabilizes token representations, and improves compositional reasoning. |
Less is More: Knowledge-Aware Compression for Long Legal Judgment Prediction (2026.findings-acl)
Copied to clipboard
| Challenge: | Recent advances leverage large language models (LLMs) for legal reasoning, but they face high computational costs and information degradation when handling long cases. |
| Approach: | They propose a framework that selectively retains legally relevant information while reducing redundant or less informative content, enabling efficient and accurate long-context reasoning. |
| Outcome: | The proposed framework outperforms existing methods on four real-world datasets spanning multiple jurisdictions and languages. |